Comparing the Performance of Popular Large Language Models on the National Board of Medical Examiners Sample Questions.

INTRODUCTION: Large language models (LLMs) have transformed various domains in medicine, aiding in complex tasks and clinical decision-making, with OpenAI's GPT-4, GPT-3.5, Google's Bard, and Anthropic's Claude among the most widely used. While GPT-4 has demonstrated superior performance in some studies, comprehensive comparisons among these models remain limited. Recognizing the significance of the National Board of Medical Examiners (NBME) exams in assessing the clinical knowledge of medical students, this study aims to compare the accuracy of popular LLMs on NBME clinical subject exam sample questions. METHODS: The questions used in this study were multiple-choice questions obtained from the official NBME website and are publicly available. Questions from the NBME subject exams in medicine, pediatrics, obstetrics and gynecology, clinical neurology, ambulatory care, family medicine, psychiatry, and surgery were used to query each LLM. The responses from GPT-4, GPT-3.5, Claude, and Bard were collected in October 2023. The response by each LLM was compared to the answer provided by the NBME and checked for accuracy. Statistical analysis was performed using one-way analysis of variance (ANOVA). RESULTS: A total of 163 questions were queried by each LLM. GPT-4 scored 163/163 (100%), GPT-3.5 scored 134/163 (82.2%), Bard scored 123/163 (75.5%), and Claude scored 138/163 (84.7%). The total performance of GPT-4 was statistically superior to that of GPT-3.5, Claude, and Bard by 17.8%, 15.3%, and 24.5%, respectively. The total performance of GPT-3.5, Claude, and Bard was not significantly different. GPT-4 significantly outperformed Bard in specific subjects, including medicine, pediatrics, family medicine, and ambulatory care, and GPT-3.5 in ambulatory care and family medicine. Across all LLMs, the surgery exam had the highest average score (18.25/20), while the family medicine exam had the lowest average score (3.75/5).  Conclusion: GPT-4's superior performance on NBME clinical subject exam sample questions underscores its potential in medical education and practice. While LLMs exhibit promise, discernment in their application is crucial, considering occasional inaccuracies. As technological advancements continue, regular reassessments and refinements are imperative to maintain their reliability and relevance in medicine.

DO seniors and IMGs have lower match probabilities than MD seniors after adjusting for specialty choice and USMLE Step 1 score.

CONTEXT: It is unknown if US residency applicants of different educational backgrounds (US allopathic [MD], Doctor of Osteopathic Medicine [DO], and international medical graduates [IMG]) but comparable academic performance have similar match success. OBJECTIVES: Our objective was to compare match probabilities between applicant types after adjusting for specialty choice and United States Medical Licensing Examination (USMLE) Step 1 scores. METHODS: We performed a secondary analysis of published data in National Resident Matching Program (NRMP) reports from 2016, 2018, 2020, and 2022 for US MD seniors, DO seniors, and IMGs (US citizens and non-US citizens). We examined the 10 specialties with the most available spots in 2022. Average marginal effects from a multiple variable logistic regression model were utilized to estimate each non-MD senior applicant type's probability of matching into their preferred specialty compared to MD seniors adjusting for specialty choice, Step 1 score, and match year. RESULTS: Each non-MD applicant type had a lower adjusted percent difference in matching to their preferred specialty than MD seniors, -7.1 % (95 % confidence interval [CI], -11.3 to -2.9) for DO seniors, -45.6 % (-50.6 to -40.5) for US IMGs, and -56.6 % (-61.5 to -51.6) for non-US IMGs. Similarly, each non-MD applicant type had a lower adjusted percent difference in matching than MD seniors across almost all Step 1 score ranges, except for DO seniors with Step 1 scores <200 (-2.0 % [-9.5 to 5.5]). CONCLUSIONS: After adjusting for specialty choice, Step 1 score, and match year, non-US MD applicants had lower probabilities of matching into their preferred specialties than their US MD colleagues.

Early use of high-efficacy therapies in multiple sclerosis in the United States: benefits, barriers, and strategies for encouraging adoption.

Multiple sclerosis (MS) is characterized by progressive neuroinflammation and neurodegeneration from disease onset that, if left untreated, can result in the accumulation of irreversible neurological disability. Early intervention with high-efficacy therapies (HETs) is increasingly recognized as the best strategy to delay or mitigate disease progression from the earliest stages of the disease and to prevent long-term neurodegeneration. Although there is growing clinical and real-world evidence supporting early HET intervention, foregoing this strategy in favor of a traditional escalation approach prioritizing lower-efficacy disease-modifying therapies remains a common approach in clinical practice. This review explores potential health care professional- and patient-related barriers to the early use of HETs in patients with MS in the United States. Barriers can include regulatory and reimbursement restrictions; knowledge gaps and long-term safety concerns among health care professionals; and various individual, cultural, and societal factors affecting patients. Potential strategies for overcoming these barriers and encouraging early HET use are proposed.

The temporal association between adverse drug reactions and antirheumatic drugs utilisation in Western Australia: a retrospective study from real-world data (1995-2015).

BACKGROUND/OBJECTIVES: Adverse drug reactions (ADRs) can result in morbidity, mortality, and higher healthcare costs. Given the limited information available on ADRs associated with antirheumatic medications, this study aims to analyse and compare ADR reporting for these drugs in the pharmacovigilance datasets of Western Australia (WA) and the United States (US). METHODS: Therapeutic Goods Administration provided WA pharmacovigilance data of selected antirheumatic drugs to from 1995 to 2015. The proportional reporting ratio (PRR) for WA case reports was compared to corresponding USA pharmacovigilance data by assessing the disproportionality of each ADR. clinically significant or true ADRs were determined using the Evans 2001 criteria (n > 2, chi-square > 4, PRR > 2). RESULTS: A total of 232 reports were found in WA, mostly on sixty-nine women aged 45 to 69. Methotrexate, leflunomide, azathioprine, sulfasalazine, and infliximab had the highest reported ADRs, related to gastrointestinal disorders. Patients who used biological agents in WA had 2.7 times the likelihood of reporting true ADRs compared to conventional antirheumatic drugs. The ADR rates in the two datasets were comparable over the study period. CONCLUSIONS: The PRR values of ADRs were consistent between WA and US databases. Methotrexate and infliximab use were commonly associated with ADR reports in WA females, with incidence rates comparable to the US; while patients using biological agents were more likely to report true ADRs than those on conventional antirheumatic drugs in WA.

Immunogenicity of quadrivalent human papillomavirus vaccine among Alaska Native children aged 9-14 years at 5 years after vaccination.

BACKGROUND: Persistent human papillomavirus (HPV) infection can cause anogenital and oropharyngeal cancers. Many HPV infections and HPV-associated cancers are vaccine-preventable. Studies suggest long-term persistence of vaccine-induced antibodies. However, data are limited among Alaska Native people. METHODS: During 2011-2014, we enrolled Alaska Native children aged 9-14 years who received a 3-dose series of quadrivalent HPV vaccine (4vHPV). We collected sera at 1 month and 1, 2, 3, and 5 years post-vaccination to evaluate trends in type-specific immunoglobulin G antibody concentrations for the 4vHPV types (HPV 6/11/16/18). RESULTS: All participants (N = 469) had detectable antibodies against all 4vHPV types at all timepoints post-vaccination. For all 4vHPV types, antibody levels peaked by 1 month post-vaccination and gradually declined in subsequent years. At 5 years post-vaccination, antibody levels were higher among children who received 4vHPV at a younger age. CONCLUSIONS: Alaska Native children maintained antibodies against all 4vHPV types at 5 years post-vaccination.

Clients' experiences and satisfaction with produce prescription programs in California: a qualitative evaluation to inform person-centered and respectful program models.

Background: Produce prescription programs have strong potential to improve food security, fruit and vegetable consumption, and health across the life course. Understanding clients' experiences and satisfaction with produce prescription programs is critical for evaluating the person-centeredness and quality of these programs. The objectives of this study were to (1) describe client experiences and satisfaction with produce prescription programs, with an emphasis on the extent to which they felt they were treated with respect and dignity, and (2) identify recommendations for improving client experiences. Methods: We conducted four focus group discussions with clients of produce prescription programs in two Federally Qualified Health Centers in California. We used a modified framework analysis approach and organized participants' experiences with programs into themes. Results: Three themes captured participants' program experiences. First, respectful produce prescription programming encompassed interactions with individuals delivering the programs that felt respectful (e.g., program staff showing they cared about participants' health and offering timely assistance with financial incentives) and disrespectful (e.g., not receiving prompt responses to questions about incentives), as well as aspects of program design perceived to be respectful (e.g., provision of gift cards as financial incentives, which offered privacy when purchasing produce). Second, having autonomy to use gift cards to choose their preferred fresh fruits and vegetables was viewed as a positive experience, though participants desired greater autonomy to shop at stores other than the program designated stores. Third, participants frequently discussed program usability, with some reporting that joining the programs and using the cards was easy, and others describing difficulties activating cards and using them at stores due to cashiers' lack of awareness of the programs. Overall, participants were highly satisfied with the programs. To improve client experiences, they recommended increasing privacy (e.g., by educating cashiers on the programs so that clients do not need to explain in public what the card is for) and autonomy (e.g., allowing cards to be used at other chain or local stores). Discussion: Our findings inform efforts to make produce prescription programs more person-centered and respectful, which in turn may increase program demand, engagement, and impact.

Immigrant-blind care: How immigrants experience the "inclusive" health system as they access care.

A growing body of scholarship examines the varying impact of legal status and race on accessing healthcare. However, a notable gap persists in comprehending the supplementary mechanisms that hinder immigrants' pathway to seek care. Drawing on ethnographic observations in various clinical settings and in-depth interviews with 28 healthcare professionals and 12 documented Haitian immigrants in a city in Upstate New York, between 2019 and 2021, I demonstrate the tension between the conceptualization and implementation of inclusive care practices by healthcare providers. I argue that the mere expansion and adoption of inclusive discourse among providers do not inherently ensure equity and the removal of barriers to healthcare access. This work contributes to the social study of medicine and race and ethnic studies by introducing the innovative concept of "immigrant-blind." Through this concept, the research sheds light on how providers' conceptualization of inclusivity proclaims medical encounters to be devoid of stratifications and rationalizes their practices which mask the profound impact of immigration status and immigration on immigrant health. Furthermore, these practices reinforce existing divisions within care settings and medical encounters, where immigration laws and enforcement practices operate and further exacerbate stratifications. By examining providers' uninformed implementation of culturally competent care practices, the findings reveal that providers stigmatize and essentialize immigrants during medical encounters. This highlights the imperative for a more nuanced and informed approach to healthcare provision, where genuine inclusivity is upheld, and barriers to access are dismantled to foster equitable and dignified healthcare experiences for all.

Perspectives from the pandemic epicenter: Sexual and reproductive health of immigrant women in New York City.

CONTEXT: The United States' response to COVID-19 created a policy, economic, and healthcare provision environment that had implications for the sexual and reproductive health (SRH) of racialized and minoritized communities. Perspectives from heterogenous immigrant communities in New York City, the pandemic epicenter in the United States (US), provides a glimpse into how restrictive social policy environments shape contraception, abortion, pregnancy preferences, and other aspects of SRH for marginalized immigrant communities. METHODS: We conducted in-depth interviews in 2020 and 2021 with 44 cisgender immigrant women from different national origins and 19 direct service providers for immigrant communities in New York City to explore how immigrants were forced to adapt their SRH preferences and behaviors to the structural barriers of the COVID-19 pandemic. We coded and analyzed the interviews using a constant comparative approach. RESULTS: Pandemic-related fears and structural barriers to healthcare access shaped shifts in contraceptive use and preferences among our participants. Immigrant women weighed their concerns for health and safety and the potential of facing discrimination as part of their contraceptive preferences. Immigrants also described shifts in their pregnancy preferences as rooted in concerns for their health and safety and economic constraints unique to immigrant communities. CONCLUSION: Understanding how immigrant women's SRH shifted in response to the structural and policy constraints of the COVID-19 pandemic can reveal how historically marginalized communities will be impacted by an increasingly restrictive reproductive health and immigration policy landscape.

Prevalence of obesity in religious clergy in the United States: A systematic review and meta-analysis.

OBJECTIVE: This systematic review aims to summarize the current body of evidence concerning the prevalence of obesity among clergy (i.e., the officially designated leaders of a religious group) in the United States. METHOD: From November 2022 to February 2023, five databases, one data repository, and gray matter were searched for articles and data sources. The search was restricted to articles published or raw data collected from 2001 to 2021. Study quality was assessed with a template, and heterogeneity was assessed using the I 2 $$ {I}^2 $$ statistic. The protocol for this review was registered with PROSPERO (CRD42022376592). RESULTS: Forty-seven studies of clergy obesity involving 35,064 individuals were eligible. The pooled prevalence estimate of obesity across studies was 34.8% (95% confidence interval [CI]: 32.5-37.2). Obesity prevalence was found to be increasing over time and to vary considerably between clergy from different religious traditions. Compared to national estimates, from 2005 onwards, obesity prevalence was higher than in the US adult population.

Lessons Learned from Cross-Systems Approach to COVID-19 Pandemic Response in Juvenile Justice System, Colorado, USA.

The global COVID-19 pandemic illustrates the importance of a close partnership between public health and juvenile justice systems when responding to communicable diseases. Many setting-specific obstacles must be navigated to respond effectively to limit disease transmission and negative health outcomes while maintaining necessary services for youth in confinement facilities. The response requires multidisciplinary expertise and collaboration to address unique considerations. Public health mitigation strategies must balance the risk for disease against the negative effects of restrictions. Key aspects of the COVID-19 response in the juvenile justice system of Colorado, USA, involved establishing robust communication and data reporting infrastructures, building a multidisciplinary response team, adapting existing infection prevention guidelines, and focusing on a whole-person health approach to infection prevention. We examine lessons learned and offer recommendations on pandemic emergency response planning and managing a statewide public health emergency in youth confinement settings that ensure ongoing readiness.

Candida auris in US Correctional Facilities.

Candida auris is an emerging fungal pathogen that typically affects patients in healthcare settings. Data on C. auris cases in correctional facilities are limited but are needed to guide public health recommendations. We describe cases and challenges of providing care for 13 patients who were transferred to correctional facilities during January 2020-December 2022 after having a positive C. auris specimen. All patients had positive specimens identified while receiving inpatient care at healthcare facilities in geographic areas with high C. auris prevalence. Correctional facilities reported challenges managing patients and implementing prevention measures; those challenges varied by whether patients were housed in prison medical units or general population units. Although rarely reported, C. auris cases in persons who are incarcerated may occur, particularly in persons with known risk factors. Measures to manage cases and prevent C. auris spread in correctional facilities should address setting-specific challenges in healthcare and nonhealthcare correctional environments.

Correlation of SARS-CoV-2 in Wastewater and Individual Testing Results in a Jail, Atlanta, Georgia, USA.

Institution-level wastewater-based surveillance was implemented during the COVID-19 pandemic, including in carceral facilities. We examined the relationship between COVID-19 diagnostic test results of residents in a jail in Atlanta, Georgia, USA (average population ≈2,700), and quantitative reverse transcription PCR signal for SARS-CoV-2 in weekly wastewater samples collected during October 2021‒May 2022. The jail offered residents rapid antigen testing at entry and periodic mass screenings by reverse transcription PCR of self-collected nasal swab specimens. We aggregated individual test data, calculated the Spearman correlation coefficient, and performed logistic regression to examine the relationship between strength of SARS-CoV-2 PCR signal (cycle threshold value) in wastewater and percentage of jail population that tested positive for COVID-19. Of 13,745 nasal specimens collected, 3.9% were COVID-positive (range 0%-29.5% per week). We observed a strong inverse correlation between diagnostic test positivity and cycle threshold value (r = -0.67; p<0.01). Wastewater-based surveillance represents an effective strategy for jailwide surveillance of COVID-19.

Development and Evaluation of Surveillance System for Identifying Jail-Associated COVID-19 Cases in Minnesota, USA, 2022.

Confinement facilities are high-risk settings for the spread of infectious disease, necessitating timely surveillance to inform public health action. To identify jail-associated COVID-19 cases from electronic laboratory reports maintained in the Minnesota Electronic Disease Surveillance System (MEDSS), Minnesota, USA, the Minnesota Department of Health developed a surveillance system that used keyword and address matching (KAM). The KAM system used a SAS program (SAS Institute Inc., https://www.sas.com) and an automated program within MEDSS to identify confinement keywords and addresses. To evaluate KAM, we matched jail booking data from the Minnesota Statewide Supervision System by full name and birthdate to the MEDSS records of adults with COVID-19 for 2022. The KAM system identified 2,212 cases in persons detained in jail; sensitivity was 92.40% and specificity was 99.95%. The success of KAM demonstrates its potential to be applied to other diseases and congregate-living settings for real-time surveillance without added reporting burden.

Outbreak of Invasive Serratia marcescens among Persons Incarcerated in a State Prison, California, USA, March 2020-December 2022.

Serratia marcescens is an environmental gram-negative bacterium that causes invasive disease in rare cases. During 2020-2022, an outbreak of 21 invasive Serratia infections occurred in a prison in California, USA. Most (95%) patients had a history of recent injection drug use (IDU). We performed whole-genome sequencing and found isolates from 8 patients and 2 pieces of IDU equipment were closely related. We also identified social interactions among patients. We recovered S. marcescens from multiple environmental samples throughout the prison, including personal containers storing Cell Block 64 (CB64), a quaternary ammonium disinfectant solution. CB64 preparation and storage conditions were suboptimal for S. marcescens disinfection. The outbreak was likely caused by contaminated CB64 and propagated by shared IDU equipment and social connections. Ensuring appropriate preparation, storage, and availability of disinfectants and enacting interventions to counteract disease spread through IDU can reduce risks for invasive Serratia infections in California prisons.

Lessons Learned from Public Health and State Prison Collaborations during COVID-19 Pandemic and Multifacility Tuberculosis Outbreak, Washington, USA.

The large COVID-19 outbreaks in prisons in the Washington (USA) State Department of Corrections (WADOC) system during 2020 highlighted the need for a new public health approach to prevent and control COVID-19 transmission in the system's 12 facilities. WADOC and the Washington State Department of Health (WADOH) responded by strengthening partnerships through dedicated corrections-focused public health staff, improving cross-agency outbreak response coordination, implementing and developing corrections-specific public health guidance, and establishing collaborative data systems. The preexisting partnerships and trust between WADOC and WADOH, strengthened during the COVID-19 response, laid the foundation for a collaborative response during late 2021 to the largest tuberculosis outbreak in Washington State in the past 20 years. We describe challenges of a multiagency collaboration during 2 outbreak responses, as well as approaches to address those challenges, and share lessons learned for future communicable disease outbreak responses in correctional settings.

HIV Care Continuum and Preexposure Prophylaxis Program in Federal Bureau of Prisons, United States.

In 2019, the US Department of Health and Human Services launched the Ending the HIV Epidemic in the US initiative (EHE) with the goal of reducing new HIV infections by 90% by 2030. This initiative identifies 4 pillars (diagnose, treat, prevent, and respond) to address the HIV epidemic in the United States. To advance the EHE goals, the Federal Bureau of Prisons (FBOP) has implemented interventions at all points of the HIV care continuum. The FBOP has addressed the EHE pillar of prevention through implementing preexposure prophylaxis, developing a strategy to decrease the risk of new HIV infection, and providing guidance to FBOP healthcare providers. This article describes the implementation of programs to improve the HIV care continuum and end the epidemic of HIV within the FBOP including a review of methodology to implement an HIV preexposure prophylaxis program.

Advancing Hepatitis C Elimination through Opt-Out Universal Screening and Treatment in Carceral Settings, United States.

Incarcerated persons are infected with hepatitis C virus (HCV) at rates ≈10 times higher than that of the general population in the United States. To achieve national hepatitis C elimination goals, the diagnosis and treatment of hepatitis C in incarcerated persons must be prioritized. In 2022, the Centers for Disease Control and Prevention recommended that all persons receive opt-out HCV screening upon entry into a carceral setting. We review recommendations, treatments, and policy strategies used to promote HCV opt-out universal HCV screening and treatment in incarcerated populations in the United States. Treatment of hepatitis C in carceral settings has increased but varies by jurisdiction and is not sufficient to achieve HCV elimination. Strengthening universal HCV screening and treatment of HCV-infected incarcerated persons is necessary for HCV elimination nationwide.

Patient experiences using public and private insurance coverage for abortion in Illinois: Implementation successes and remaining gaps.

CONTEXT: Insurance coverage for abortion in states where care remains legal can alleviate financial burdens for patients and increase access. Recent policy changes in Illinois required Medicaid and some private insurance plans to cover abortion care. This study explores policy implementation from the perspectives of patients using their insurance to obtain early abortion care. METHODOLOGY: Between July 2021 and February 2022, we interviewed Illinois residents who recently sought abortion care at ≤11 weeks of pregnancy. We also interviewed nine key informants with experience providing or billing for abortion or supporting insurance policy implementation in Illinois. We coded interview transcripts in Dedoose and developed code summaries to identify salient themes across interviews. RESULTS: Most participants insured by Illinois Medicaid or eligible for enrollment received full coverage for their abortions; most with private insurance did not and faced challenges learning about coverage status. Some opted not to use insurance, often citing privacy concerns. Participants who benefited from abortion coverage expressed relief, gave examples of other financial challenges they could prioritize, and described feeling in control of their abortion experience. Those without coverage described feeling stressed, uncertain, and constrained in their decision-making. CONCLUSION: When abortion was fully covered by insurance, it reduced financial burdens and enhanced reproductive autonomy. Illinois Medicaid policy-with seamless enrollment options and appropriate reimbursement rates-offers a model for improving abortion access in other states. Further investigation is needed to determine compliance among private insurance companies and increase transparency.

Exploring kidney dialysis costs in the United States: a scoping review.

AIMS: The increasing prevalence of end-stage renal disease (ESRD) in the United States (US) represents a considerable economic burden due to the high cost of dialysis treatment. This review examines data from real-world studies to identify cost drivers and explore areas where dialysis costs could be reduced. METHODS: We identified and synthesized evidence published from 2016-2023 reporting direct dialysis costs in adult US patients from a comprehensive literature search of MEDLINE, Embase, and grey literature sources (eg, US Renal Data System reports). RESULTS: Most identified data related to Medicare expenditures. Overall Medicare spending in 2020 was $29B for hemodialysis and $2.8B for peritoneal dialysis (PD). Dialysis costs accounted for almost 80% of total Medicare expenditures on ESRD beneficiaries. Private insurance payers consistently pay more for dialysis; for example, per person per month spending by private insurers on outpatient dialysis was estimated at $10,149 compared with Medicare spending of $3,364. Dialysis costs were higher in specific high-risk patient groups (eg, type 2 diabetes, hepatitis C). Spending on hemodialysis was higher than on PD, but the gap in spending between PD and hemodialysis is closing. Vascular access costs accounted for a substantial proportion of dialysis costs. LIMITATIONS: Insufficient detail in the identified studies, especially related to outpatient costs, limits opportunities to identify key drivers. Differences between the studies in methods of measuring dialysis costs make generalization of these results difficult. CONCLUSIONS: These findings indicate that prevention of or delay in progression to ESRD could have considerable cost savings for Medicare and private payers, particularly in patients with high-risk conditions such as type 2 diabetes. More efficient use of resources is needed, including low-cost medication, to improve clinical outcomes and lower overall costs, especially in high-risk groups. Widening access to PD where it is safe and appropriate may help to reduce dialysis costs.

Previous papers have studied the cost of treating patients who need dialysis for kidney failure. We reviewed these costs and looked for patterns.Dialysis was the most expensive part of treatment for people with kidney disease who have Medicare. Dialysis with private insurance was much more expensive than with Medicare. People with diabetes experienced higher costs of dialysis than those without diabetes. Dialysis in a hospital costs more than dialysis at home.There are opportunities to reduce the cost of dialysis that should be explored further, such as more use of low-cost medication that can prevent the worsening of kidney disease and reduce the need for dialysis.

Adopting the RE-AIM analytic framework for rural program evaluation: experiences from the Advance Care Planning via Group Visits (ACP-GV) national evaluation.

Introduction: To support rigorous evaluation across a national portfolio of grants, the United States Department of Veterans Affairs (VA) Office of Rural Health (ORH) adopted an analytic framework to guide their grantees' evaluation of initiatives that reach rural veterans and to standardize the reporting of outcomes and impacts. Advance Care Planning via Group Visits (ACP-GV), one of ORH's Enterprise-Wide Initiatives, also followed the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. ACP-GV is a national patient-centered intervention delivered in a large, veterans integrated healthcare system. This manuscript describes how RE-AIM was used to evaluate this national program and lessons learned from ORH's annual reporting feedback to ACP-GV on their use of the framework to describe evaluation impacts. Methods: We used patient, provider, and site-level administrative health care data from the VA Corporate Data Warehouse and national program management databases for federal fiscal years (FY) spanning October 1, 2018-September 30, 2023. Measures included cumulative and past FY metrics developed to assess program impacts. Results: RE-AIM constructs included the following cumulative and annual program evaluation results. ACP-GV reached 54,167 unique veterans, including 19,032 unique rural veterans between FY 2018 to FY 2023. During FY 2023, implementation adherence to the ACP-GV model was noted in 91.7% of program completers, with 55% of these completers reporting a knowledge increase and 14% reporting a substantial knowledge increase (effectiveness). As of FY 2023, 66 ACP-GV sites were active, and 1,556 VA staff were trained in the intervention (adoption). Of the 66 active sites in FY 2023, 27 were sites previously funded by ORH and continued to offer ACP-GV after the conclusion of three years of seed funding (maintenance). Discussion: Lessons learned developing RE-AIM metrics collaboratively with program developers, implementers, and evaluators allowed for a balance of clinical and scientific input in decision-making, while the ORH annual reporting feedback provided specificity and emphasis for including both cumulative, annual, and rural specific metrics. ACP-GV's use of RE-AIM metrics is a key step towards improving rural veteran health outcomes and describing real world program impacts.